Port bandwidth leak prevention: backend degradation detection & gating#182
Open
full-bars wants to merge 1 commit into
Open
Port bandwidth leak prevention: backend degradation detection & gating#182full-bars wants to merge 1 commit into
full-bars wants to merge 1 commit into
Conversation
I've ported the backend degradation detection and contract gating mechanisms from my [v3.23-fix implementation](https://github.com/full-bars/urnetwork-3.23-fix) to upstream. This implementation distinguishes sustained API outages from transient timeouts, then gates contract creation and applies exponential backoff to resend queues when degraded. This prevents providers from wasting bandwidth on contract attempts and retransmissions during control API failures, effectively reducing the bandwidth leak during outage windows. The mechanism was validated in production during a recent incident across ~120 servers with 11 minutes of outage data. Changes: - transport.go: Backend degradation detection atomics, threshold logic, auth error tracking - transfer_contract_manager.go: OOB error failure tracking and reset on success - transfer.go: Contract retry backoff, contract creation gating, resend exponential backoff
Author
|
@xcolwell — requesting review on this bandwidth leak prevention PR. This ports the backend degradation detection and contract gating mechanisms from the fork I maintain to upstream, addressing issues #175 and #181. The implementation distinguishes sustained API outages from transient timeouts, then gates contract creation and applies exponential backoff when degraded. Validated in production during a recent 11-minute control API outage across ~120 servers—bandwidth consumption dropped to ~12% of baseline, confirming the mechanisms effectively reduce the leak. Related: PR#180 (log spam fixes for auth/contract/drop errors). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bandwidth Leak Prevention: Backend Degradation Detection & Gating
I've ported the proven backend degradation detection and contract gating mechanisms
from my v3.23-fix implementation
to upstream. This addresses the data leak during control API outages (issues #175, #181).
The Problem
When the control API is unreachable, providers continue attempting contract creation
and queueing transfers against a dead API. Without throttling, this becomes a massive
retry storm: thousands of route attempts per second, each spawning goroutines and
queuing transfers that will never complete because clients can't authorize.
The provider essentially screams at an API that isn't even listening, blasting the
network with data that has no chance of reaching a client. On metered bandwidth
connections (common for proxy providers), a sustained outage can consume an entire
month's bandwidth allocation in just a few hours. For proxy users relying on that
bandwidth, this translates to service loss beyond the API outage itself — the pipe
is wide open but the data is being wasted.
The core issue: there's no signal to the provider that the API is down. So it keeps
trying, keeps transferring, keeps wasting bandwidth, until either the outage ends or
the budget is exhausted. This fix adds that signal via degradation detection, then
uses it to stop the bleeding.
The Solution
Backend Degradation Detection:
positives from transient timeouts
Gating & Throttling When Degraded:
intervals instead of sending them in quick succession)
Production Validation
I validated this strategy during a recent control API outage on my fleet (2026-05-30, 3:10-3:21 PM PDT):
that the mechanisms effectively reduce (not eliminate) the leak
intended cleanup behavior (clients and contracts are long gone, so it amounts to
clearing queues). The strategy remains effective — the leak is contained during the
actual outage window.
Important caveat: An 11-minute window provides limited statistical power. The data
shows the mechanism works as designed, but edge cases or longer outages might reveal
new issues. The fix is based on production code that's been running in my
v3.23-fix implementation, so
confidence is high.
Implementation Details
transient timeouts
feedback loops
Related PRs